Discussion of “Spline Adaptation in Extended Linear Models”, by Hansen and Kooperberg

This paper uses ideas for stochastic search implementations of adaptive Bayesian models, such as those outlined in Denison, Malick and Smith (1998 a,b) and Chipman, George and McCulloch (1998a) and effectively applies these ideas to logspline density estimation and triogram regression. Interesting comparisons are made to assess the effect of greedy search, stochastic search and model averaging. Such comparisons are valuable, since readily available computing power enables the construction of many methods, and an understanding of what works is important in developing new methodology. It is very important to note the role of the prior when adaptive models are used in conjunction with stochastic searches. Inevitably, priors guide and temper our wandering in a large space of models. This benefit comes with a price: the need to select a prior that is appropriate for the problem at hand. It is important to acknowledge the simple fact that a prior choice represents a bet on what kind of models we want to consider. If we skip to the end of the paper and read the discussion, what lessons have been learned? We have (i) ”.. we have demonstrated a gain .. when appealing to the more elaborate sampling schemes” (relative to simple greedy search), and (ii) ”priors play an important role”. These things we know to be true in general from much experience. The question is: what should be done in practice? In general, a practical approach usually involves first getting the prior specification down to a few hyper-parameters (about which we hopefully have some understanding) and then developing a scheme for making reasonable choices. At one end of the spectrum we can use automatic methods such as cross-validation to choose hyper-parameters that are appropriate for the problem at hand. At the other end of the spectrum we choose ”reasonable values” based on our understanding and prior beliefs. Often, compromise strategies that combine a peek at the data with some judgment are effective and somewhat in the spirit of empirical Bayes. We believe Chipman, George, and McCulloch (2002) is a good example of this middle ground approach. We have some general Bayesian insights that help us understand the effects of these hyper-parameters. Often we can think of prior in two stages: p(Mk) a prior on ”models”, and p(θk|Mk) a prior on the parameters of a given model. A set of hyper-parameters would specify a choice for each of these components.